2023-11-06
Create a scanner which can collect local population data outdoors ✅ and indoors 🚧
Get people to walk across the scanner 😬
Collect large amounts of shoe pictures w/ time, date, location data 🥿👠👞
Create an algorithm which can identify human-recognizable characteristics from scanner images
Generate a local database of common footwear features
Use local database to characterize frequency of similar shoes in local population
We can reasonably pose this problem in 3 different ways:
Each method requires a different type of labeling schema, data format, etc.
Some are more tedious to generate than others
Use VGG16 to classify 256x256 px chunks of images
Goal is to label the entire chunk with one or more classes
VGG16 Shoe Example approach
Try to use object detection
Started with FastAI, but had terrible ongoing support
Moved to PyTorch, which is a lower level framework
(Side note: Muxin is amazing at Python)
New developments:
Structured model so that the underlying network was replacable: can swap VGG16 for Resnet50
Implementing better metrics, e.g. Intersection over Union for assessing predictions vs. ground truth
Neural networks are trained on millions of human-annotated real-world photos
Even shoe soles are artificial relative to a natural scene
Networks weren’t trained on the artificial patterns or layouts that are used to generate shoes
Pretrained NNs can still generate useful information that is computer-friendly (e.g. Charless’s method)
To work with artificial patterns and get human-like labels, we have to do something different.
Systematically generate a large library of synthetic data
pre-labeled
complex characteristics
will require several iterations to get right
Use to train preliminary model
Run 2D patterns through Charless’s network to generate more realistic 3D images
Run 2D patterns through Charless’s network to generate more realistic 3D images
Train on Zappos pictures labeled by humans
Train on Scanner Photos
Measure performance/accuracy changes over time on a consistent set of stimuli manually derived from real shoes
SVGs can include metadata
Easy scaling
SVG intersection will allow marking partial objects
Region segmentation + image labels
Manual SVG creation (8h \(\approx\) 52 images)
New R/python library to generate data
3D rendering after 2D stage:
Lots of work required before we start in on photos